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ABSTRACT 

In response to a statewide 
evaluation, the^ State of Illinois funded a 
evaluation project. Developed by personnel 
Illinois and Northern Illinois University, 
toward devising and delivering informaition 
education agencies' (LEA's) to h^'lp improve 

efforts- Project objectives were to: (1) identify local and state 
information needs, (2) develop eclectic evaluation strategies which 
meet the needs of local and state agencies, (3) field-test strategies 
in selected LEA'S, and (U) disseminate selected locally-directed 
evaluation strategies to key administrators in LEA"s via. workshops, 
(Author) 
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RATIONALE 

i 

Within the past centQry over 25 bilMon dollars has been expended by 
federal agencies alone in the Improvement of educational programs in 
elementary and secondary schools, colleges, and un^^rsTti^s, A recent 
report to Congress and the President by the National Advisory Council on 
'Education Professions Development indicates a change is occurring by con- 
cluding "the emphasis has shifted from the initiation of efforts aimed at 
bringing about improvements in American education to an interest In deter- 
mining whether these efforts are achieving their goals.'* (NACEPD, 197'+) 
The importance of evaluation is beifig realized at all levels including 
national, state, and local. 

National Emphasis ^ 

At the national or federal level, eva 1 ua 1 1 on .has been emptiasized as 
far back ^s the Vocational Education Act of 1963 and the Elementary and 
Secondary Education Act of 1965. The Advisory Council for Education 
Professions Development has been conducting an inquiry into the way federal. 
programs--and state and local programs supported by federal funds--are being 
eval uated • 

Also in their 197^ report, the NACC?D cites four developments which 

give an Indication of the importance attached to evaluation at the national 

level. The four developments include: 

K Increasingly in new legislation enacted.-and in amendments to 
existing statutes, explicit provision for evaluation has been 
made. The General Accounting Office reports that in the areas 
of health, education, welfare, economic opportunity, and like 
federal activities, there are twenty-three Acts in which formal 
objective evaluation is now prescribed. Some of these statutes 
have numerous provi s I ons for eva 1 ua t i on . 
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2. The Executive Branch has been placing increasing emphasis on^ 
"cost-effectiveness*' as a means for determining whether the 
results from given Federal prcJgrams are in proportion to the 
money expended. Such a procedure assumes the capacity on the 
part of the agencies to#carry dut appropriate and effective 
eval uation. 

3. There has been a very substantial increase i*n the- funds allotted 
for evaluation in the Executive Branch, The General Accounting 
Office reports that expenditures for non-defense program evalua- 
tion have jumped from $19 million in I960 to $110 million In 
1972. • . 

k,' I t Ks clear that before the end of this year the Congress will 

have instituted some major reforms related to fundamental aspects 
of its own procedures. We speak of provisions for the establish- 
ment of budget ceMH ngv ; for machinery which will enable the 
Congress tb set broad areas of prior i ty concern; and for the 
creation of a Congressional Office of the Budget. The success 
of these reforms will depend, In ho small way, on the capacity 
of the Congress to evaluate the effectiveness of the program it 
has legislated — or at least to interpret evaluations conducted 
by the Executive Branch and to appraise the quality of such 
eval uat i ons . 

Obviously, the concern for evaluation by our nation's leadership has 
been considerable. But the success of most or many federal evaluations 
rely on the adequacy^ of evaluation activity at the state and local level. 
State Emphasis 

Evaluation efforts at the state level have lagged behind their federal 
mandates in almost all states. However, some states have realized the need 
and importance of evaluation and have designed systems for evaluating educa- 
tional programs. The Illinois Division of Vocational and Technical EducaA^on 
has been a leader in the development and implementation of a statewide 
evaluation system. Many other states have since initiated evaluation sys- 
tems and many have relied on the expertise and experience of Mlinois per- 
sonnel. 

Loca 1 Emphas i s 

During FY 72 jnd FY 73, a major finding of Illinois' evaluation system 
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for the statewide'' evaluation of occupational programs was that there has 



existed a serious lack of development and utilization of evaluat^ion pro- 
cedures at the local level. Almost universally, evaluation reports of 
visitation teams stressed the need for improvement in both the development 
and refinement of local evaluation procedures and for increased emphasis 
on evaluation in pre- and in-service higher education programs. A survey, 
of local e\?aluation activij^ies Jn otherxstatps Sas led to a similar finding. 
The Illinois cc ppsite evaluation reports -j^ both FY 72 and FY 73 

(OVTE, 1972 and 1973) emphasized this f i nd i ng example conclusions from* 

this report include: * • 

Local school, personnel need assistance if they are to 
acqui re the techniques and competencies necessary to 
generate and maintain strategies for systematic evalu* 
ation.' (FY 72)" , 

Impetus in developing local educational agency '(LEA) 
evaluation systems and using their results is badly 
needed. O^Y 72) 

I ns ti tut ions for professi onal development have not 
provided training in conducting locally-directed 

evaluations. This is evidenced by the lack of for- i 
mali'zed systems at the local level. (FY 72) 

. Data gathered and interviews conducted indicate that 
very few, schools have a locally-directed evaluation 
system for their occupational program--90% of the 
schools evaluated were reported to have no locally- 
directed evaluation system. (FY 73) 

Those few LEA's who report having i ocal ly-di rccted 
evaluation systems make only limited use of the data 
generated by those jSys tems in planning for their 
occupa t i ona 1 programs . (FY 73 ) 

In response to this need, the Illinois Divisiorf of Vocational and 
Technical Education Initiated activity in devising and delivering information 
and assistance to local school evaluators through the initiation of a develop- 
mental project. 
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The gun 1 for the 1 oca 11 yd i rec ted eval ua 1 1 on projec t is to i mprove 
educational offerings \yithin local education agencies (LEA's) through 
more effective use of evaluation procedures. Locally-directed evaluation 
has been defined as any activity which LEA personnel ^^nduct to aid in 
planning and dec i s iqn-maki ng-. A more immediate goal of the project was; 
to systemize several of these activities for use by 'LEA personnel —givi ng 
them practical guidelines to the conduct of each form of evaluation they 
would be llkel.y to encounter* To further delineate the project goal and 
give direction tc the project; five project object i ves were formulated: 

1. J& identify local and state agency needs for evaluative 
information, 

2. To develop a practical strategy for locally-directed 
eva 1 ua t ion ,bas i ng it on current theory, 

3. To develop practical evaluation activities to beutilized 
^ by LEA personnel , 

^. /Jo field test the evaluation strategy and activities in 
selected LEA's, an<J 

5. To design a scheme to disseminate and encourage imple- 
, mentation of locally-directed evaluation activities. ^ 

Procedures 

Ba^ed upon the aforementioned goal end objcclives, a project plan was 
formulated. This plan included a number of procedural steps which are 
briefly described below. 
State and Local Needs Assessment 

The actual need for evaluation activity ai the local level was vividly 
portrayed In the year-end report of the s tote-di rected on-site evaluations* 
However, it was necessary to obtain additional information regarding thi; 



identification of existing activity, the LEA perceived need for evaluation, 
and the constraints which had de ter red eva 1 ua I i on in -the past. To aid In 
the identification of these factors, a needs assessment workshop was held 
in which thirty practicing educators par t i ci pated • The participants In- 
cluded superintendents, principals, community college presidents and deans, 
and local vocational directors. The input of all of these groups was 
gathered over a two-day period. 

I ... 

An additional concern in the needs assessment was the determination 

• * 

of state agency information needs which could be met by local sources. 
Thi^ determination was accomplished through the contact and interview of 
staff of the Professional and Curr i cu 1 un ovelopment Unit, Research and 
Development Unit, and the Program Appr and Evaluation Unit of the 

I 1 1 i noi s State Of f ice. 

A third aspect of the needs assessment involved a group of fifteen 
incjividuals who -were team leaders for. the state-directed Evaluation system. 
These* individuals were assembl ed. for a two-day meeting which has two pur- 
[H)se^: 1) to Identify local needs, and 2) to begin development of an 
evaluation straiegy. The fir^t purpose was met by dividing the group in 
half and diking each oroup lo answer the following qucslioHs ^i;r each of 
eight program areas: 

L To what extent does evaluation happen now? 

2. What would be ideal evaluation practices? 

3. What are the limits and constraints to reaching the ideal? 

What are suggested means for overcoming the discrepancy between 
current and ideal evaluation practices, in light of these con- 
straints? 

The second purpose, ihc initial attempt to develop an evaluation strategy 



or f raruewof> for local agency use, was accomplished by utilizing the some 

two groups in an advocate team approacn. The two teams were directed to 

... ■ • % 

develop conipeting strategies for loc'ol 1 y-di rec ted evaluation, A written 
and oral charge was made to the teams with specific boundaries or guidelines 
to be fol lowed • 

Revi ew of Eva! uat ion Li teratv* 



An extensive review of the literature was conducted to identify both 
theore,t i<:a1 models of eval uat i -^n and practical applications in local edu^a- . 
tion agencies. A computer search of . the ERIC system and ^the Current Index 
to Journals in Education was made, l.n addition, convention and conference 
papers dealing with evaluation were obtained for the most recent two-year 
period. Many fragmented or individual evaluation activities were identified 
through this review such as student follow-up surveys, cost analysis, etc. 
However, few were presented in a practically implementable format and. fewer 
were presented in the form of a system. In addition, many were very speci- 
fically focused in individual '*title'* projects and little attention was 
given to the general i zabi 1 i ty of activities to other educational endeavors. 
Compilation of List of Evaluation Activities 

A synthesis of the local leader needs assessment, the findings and 
suggestions of evaluation team leaders, and the review of evaluation litera- 
ture resulted in a tentative listing of thirty-eight evaluation activities. 
This listing incorporated four segments: (1) activity title, (2) faformation 
source, (3) technique, • and (^) purpose. An example entry-of this listing is 
presented below. 

PEST COPY AVA([/.S!£ 
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Activity Title 



^ The E mployer 
Survey 



Information Source 

Ennployers of former 
students 



Techniques 

MatI Survey 
Personal interview 
Telephone inter 
view 



Purpose 

DeternDine performance of former students 
Assess competency or objective lists 
Elicit employer recommendations 
Estimate supply and demand 
Determine recruitment practices 
Aid public relations 



ERIC 



This activity listing was later used in formulati nc^a , strategy or structure 
for Ipcally-di rected evaluation and in the later prioritization of activities 
to aid in developnjent decision making. - ^ ^ 
Formulation of a Structure f or Locally-Directed Evaluation 

As indicated previously, the literature review included the review of 
evaluation model s . and conceptual frameworks^ One of these models, the CIPP 
n.odel, was identified as being most appropriate to meeting the needs of the 
lea's and to. the facilitation of an on-going or continuou.s^ e.valuation effort. 
Also, it was thought, the CIPP mo4el could be utilized with any program, 
regardless of its phase of development or implementation. 

Even though the CIPP mbdel was utilized, it was determined not to des- 
cribe and define the model to LEA personnel as Stuff lebeam, et al . might have 
suggested. Instead, the components of the CIPP model were defined in the 
form of questions: * * 

Context Evaluation: Where are you? 

Where are you going? 

Input Evaluation: ' How wi 1 1 you get there? 

Process Evaluaiion: How are you doing in getting there? 

Product Evaluation: Have you made it? 

.The planned evaluation act i vi ty, "The Employer Survey, •'was easily 
derived from one of the CIPP-related evaluation questions; this activity 
was clearly a form of product ev/aluation. Projected activities pertaining 
to context evaluation included: a job need survey, assessment of communj ty 
and school resourcess, a student interest survey. Similarly, example 
activities for process evaluation included: the survey of current students, 
team review, and staff evaluation. input evaluation included activities 
such as cost analysis, and team review of alternatives. 
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Development of an Introductory Activity 

The next step in the development of th6 1(Kti1 1 yd i rccted evaluation 

project wab the development of on introductory activity* It was-felt 

<• • • • * 

that the development of such .an activity would provide, local personnel 

with a, concise introduction to the structure for locall y-di reeled evalu* 

at Ion. ^ . ^ 

The introductory blide/tape gave a quick introduction to the CTPP 

theoretical model --although it did not, for ,prev i ous ly mentioned reasons,- 

{ 

mention this model by name. Jhis introductory activfty i.n the form of a 
local leader guide and a slide/tape presentotion did introduce the viewer 
t(^ the basic questions as soc i a te^d wi th ^ the respective stages of the CIPP 
model. Although it did not list all possible activities (a number of these 
were no<t yet determined), it sugges tod ac t i v i t i es to correspond to each 
question. 

The introductory blide/tape was accompanied by a set of materials which 
w<:tjld be distributed to selected personnel within each LEA. The materials 
Included: - a local leader guide, a consultant guide, and a catalog of 
evaluation activities. ^ . . • . 

The local leader guide tliat accompanies the slide/tape presentation 
provided a set of specific procedures, '«ii.i|ilt forms or documents, a written 
copy of the script, and included reproduced copies of the slides themselves, 
Tlii^ companion wa.s compiled to provide local leaders and others with a quick 
reference to the areas of the slide/tape which they felt the need to review. 
Develop a "Pro to t ype' Ac t i v i ty 

Concurrent with the development of the introductory activity, an 
activity entitled, "Tlie Employer Survey," was developed. It was intended 
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that thi s^acti vi ty serve as a prototype to be subjected .to close scrutiny. 
It wiJs h»ped, that this activity would- allow lEA personnel to react to 'the 
format and met^iod of di ssejjni nation to be utilized as well as clrfrjfy,-by 
example, some of the Material presented in the introductory activity. The^ 
prototype i^ctivity, 1 ij<e the introductory activity, consisted of a slide/ 

^ ' '\ ^ • • • . * r 

tape presentation augmented by a printed locals leader guide, consultant 

guide entry and catalog entry. . 

Establishment of an Admi ni strati ve' Advi sory Committee 

A group of eleven LEA. chief school administrators was formed to aid the 
project staff by reacting to and making suggestions regarding project acti- 
vities. The two prototype activities we re* submi t ted to the .group for. their 
C'ons i derat i on. This group was a1 so, asked to field, t«»st the two "prototype*' 
activities in their own local agency. To achieve a balance .in the final 
product that would guarantee its usefulness in all Illinois schools, th^ 
tear was composed of administrators from schools ranging in size and geo- 
graphical area. * The administrators were cautioned not to accept the deci- 
sirtn to develop slide/tape presentations and accompanying documents as final 
for the entire series of activities. They v/ere encouraged to criticize and 
renovate the ac t i v i I i e'^ --and to suggest that* the project be terminated \S 
they found it seriously wanting or unnecessary. 

As a result of the meeting of tjie advisory co^imittee, the project was 
continued and the development a»f two more prototype activities was initiated. 
The administrators were in agreemeM concerning several important recommend- 
•at ions for the future conduct-^of the. proj ec t • The admi ni s^t rators were 
sciti'fiod with I hp baMc format of the evaluation activities, and re^commended 
thcjt the locri 11 y-di ccc torf evalaation project continue in the format in which 
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it vvos desiqncrl, The*group further agreed to participate during the 
romdinder of iKo 73 -7^ ' •>chor) 1 year to'the < > tent Jpobs i bl c ; dfter this 
Initial expori(Mico it \;a*^ fell <i decision could be re.'iched as to their 
conti nued part i cipol during the 1'97^"75 school year. Participation 
,during 73-7^ -^cnnt agreemcrut to pilot test the activity, *'The Development 
of a Locally-Directed Evaluation Plan,'* with the assi s tance of project 
staff. Adini ni b t rators were also committed to par t i c i pa t i on in a meeting 
to determine Ihe course of development' for future slide/tape presentations 
Field Test of Prototypes and Activities • 

• Field testing occurred in two phases. The first phase included the 
Ic^Ming {^J tfie introductory und employer survey activities and "the overall 
fofmaj^of activities, while the second phase was focused on th'e tesljng 
and refij-ement of the ac t i v i t f es ' desc r i bed in the following section. 

Upon completion of ihe two prototype activities a field test was 
plnnnc^d in tho'^e LEA'^ v/ibiing to participate. Schools wishing to parti* 
ci[N'^tc wore oiveti both frecdon and encouragement to apply 'the evaluation 
a( t i V I t i e* t(. I hei r f)ec i f i c s i luat i on*^ . I f a v chool wi shed tO' It^mpl oy one 
.utivilv in cmc departntnt nnd dt^sired that another activMy be conducted- 
fi c';rplicd (i i M r i c t -\; i dc , it fc^ll t hti t ihi* would work lo increase the 

ut i 1 i.l y of t he f ic 1 d tc • I . 

It wa^ thr (Ir'in of nnv chief •tli^ol .-rltiii n i s i ra tor thai (^ne (Kcupa- 
llcrr.cJly tiicntcd tictivity l;( altered h> thi ^cIk)o1 and applied to ev£^^lua- 
tion of academic programs. Thi^, too, was encouraged as the p'rojcct staff 
felt that their roMe wc»^ lo^^id local district;: in cvc- 1 ua t i on-^no I to-pro- 
wM) t c t he mc t i V i I V i t - e H , 



Identification of Needed Act ivi ties 

B^sed on the testing of the i ntroductofuy activity and the resulting 
evaluation plans made up by local sdiools, a set of, priorities was deter- 
mined for the development of future activities. Schools were given a set. 
of possible activi ties-«activi ties which it was planned would b^^developed 

if the necessity of development was confirmed by ^the school personnel. From. 

\ ' 

Jthis list, they put together an evaluation plan for the institution or dis- 
trict. Educators were encouraged to suggest additional activities which 
would fit their, needs. Tlie project staff then drew up a schedule for 
development of activPlies 4hal would put .the desired activity in the hands 

. of each district within the smallest tirhe period possible/* 
Development, of Activities 

To insure quality and to minimize the time necessary to develop act- 



ivities designated as high priority, development consultants were identified 
with expertise in areas relating to the planned act ivi t i es . In this way 
several activities could be deve loped^at one time. Consultants were first 

briefed at an orientation meeting. D.ring this meeting, they were shown 

..... 

prototype activities to familiarize them with the considerations unique 
to the slide/tape medium. At this time the group was also given more per- 
sonal guidelines and more specific advice on activity development. 

To insure uniformity in quality and format, tlie consultants were asked 
to^ complete their task in two staget. The first stage involved the outline 
of matter ial to bo presented within tht ilide/tape pKicntotion and the 
local leader guido; thii outline uti\ uviewed by tin project *. talf. If 
maieriaJ submitted v^a* t(^o cmnplcx, to(; r Uincntary > to timply irrelevant, 
consultants were asUd to r( \ i .c th(. (nit line. Once preliminary outlines 
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were^ approved, consultants were free tr complete the second stage--the 
actual development of a slide/ tape script and related document^* Slide/ 
tape scripts were again reviewed, edited ac revised' by project staff and 
forwarded to commercial production consultants. Artwork and narration 
was completed and slide/t^pes were once more reviewed and revised, if 
necessary, -prior to duplication for dissemination to the LEA's. 
Design a Dissemination Scheme 

The field test experience of the project staff not only led to the 
refinement of individual evaluation activities, but also heljied to illum- 
inate the constraints and limitations faced by locaT school personnel In 
terms of implementing evaluation activities. Better understanding of 
these cons trai nts"^ led to the development of ,a flexible scheme for imple* 
menting the outcomes of the locally-directed evaluation project* 

Basically, the di s'*semi nation plan incorporates means by which individual 
activities may be handled: 1) independently and wi thout outside assistance, 
2) with the aid of a state department consultant, or 3) with the aid of a 
privately contracted consultant. To aid consultants, when the consultant 
approach is chosen^ specific guidelines were formulated by project staff. 
These guidelines provide assistance to the consultant in introducing the 
activities to local staff, acting as a resource person in planning, reacting 
to LEA progress and Instrumentation, and aiding the LEA personnel in utilizing 
the results of the activities. 

Outcome of Project 

The outcomes of the local ly-dl rected, evaluati on project will comprise 
<^ series of evaluation activities and related documents which have been 
designed ;to help achieve the project goal. Each of the planned evaluation 
activities in this series follows a similar format. That is, each activity 
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which Is designed to aid the local leader and his staff In do! ng eval uation 
will include a slide/tape presentation, a local leader guide, a consultant 
guide, and an entry in a cfatalog of evaluation activities. 
Slide/Tape Presentations 

A sli'de/tape presentation has been or will be developed to accompany 
each evaluation activity. The first activity, "An I ntroduction»to Locally- 
Directed Evaluation," was described in a previous section. -This introductory 
activity was designed to dispell some of th^ misconceptions of evaluation { 
which have been held by educators and to orient local school personnel to 
the concepts related to locally-directed evaluation. An additional objec- 
tlve of this Introductory activity was to facilitate the development of a 
local evaluation plan for Wch„LEA. LEA plans are meant to reflect the 
concerns and problems of LEA personnel ^and allows for the .selecti on and 
sequencing of evaluation activiti-es which can be utilized to better under- 

Stand the concerns and help .overcome the identified problems. 

? 

The slide/tape presentations which accompany all other activities are 
more instructional and more directive with regard to oubtasks which must be 
accomplished. The presentations begin with a listing and discussion of the 
reasons for conducting the specific activity. For example, In the activity 
entitled, "The Employer Follow-up Survey," purposes or reasons for conducting 
the activity include: >^ 



WHAT CAN THE EMPLOYER FOLLOW-UP DO TOR USt 

t 

• determtn* ^rformanct of former students 

• assess competency or objective lUtt 

• eltctt employer recoffmendetlons ^ 
e make program comparisons 

e estimate supply, 4nd demend 

• determine recruitment practices 

• aid public relations 



Purposes such as these are then discussed and the methodological consider- 
ations of the activity^are addressed. Specif i-c attention is paid to identi- 
fying tasks which must be completed and the offering of suggestions and 

** ' * ■ . 

directions tor, tasit achievement. The slide/tape presentations are con-, 

eluded with suggestions for utilizing the obtained results in meeting the 
needs oS the lEA. 
. Local Leader jGuides 

A printed booklet accompanies each slide/tape presentation. This 
booklet is designed to augment the audio-visual pr.esen tati on for the local 
leader. The first part of each booklet contains a set of procedures which 
the local leader should follow in the planning, initiation, implementation 
and follow-up of the activity. When feasible, procedures are subdivided 
into subtasks and suggestions are given relative to the division of labor 
for the LEA staff. . For example, suggestions are given as to what types of 
individuals sljould be responsible for instrument development for the employer 
survey. 

A second section of the local leader guides includes copies of pre- 
vious ly .developed instruments, letters and guides which might aid the 
local leader in completing the evaluation activity. These appended docu- 
ments are not intended to be .used verbatim in most c.ases but instead are 
presented as examples or documents which can be adapted to the specific 
needs of the LEA. 

The last segment of each local leader guide contains a pri nfecf photo- 

. ' • •• ♦ 

graph of each slide used in the sMde/tape presentation and the tape script 

is also provided. This material is included for reference purposes and to 

avoid having to review the entire slide/tape to gain clarification of a 

point which may be needed. , - 




Consul tant Gui de 

The document referred to as the consultant guide has the purpose of" 
giving specific procedures for a private or state office consultant to 
follow in facilitating the evaluation activities within LEA's. An LEA 
within the state may request that a consultant be sent to work with the 

t 

local leader in pVannirtg an activity and orienting ^he entire staff to the 

activity^. This task is usually complemented by the slide/ tape presentation. 

Following the orientation of ,st^ff and the initial planni ng phase, the conr 

sultant serves as a resource pePion, reacting to locally produced documents 

and procedures. This last responsibility may continue over an extensive 

period of time. The consultants guide contains information useful to the 

consultant in his facllitative role. • , ' 

% • 

Catalog of Evaluation Activities 

A catalog of activities represents yet another document related to 
locally-directed evaluation, this catalog possesses an entry for each 
activity and somewhat parallels a college catalog. For each acti vi ty , the ' 
fol lowi ng I tems are included: activity title', general activity purpose, 

activity goals, prerequi si te acti vi ties , requi red- material s and estimated 

* _^ • 

•time requirements. 

*. * • * 

All of the materials mentioned in this section have been developed to 

aid the leaders within LEA's to conduct evaluation activities. The material 
along with the consul tati ve. assi stance are by no means externally imposed 
by th3 state education agency—they are strictly for local lyplanned pur- 
poses. . ; 

•J , 

Completion of 18 activities is planned for early fall of this year; 
wbile the. field testing is schedules to be complete by the end of* the 197^- 
75 school year. The original focus of the locally-directed evaluation pro- 
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ject was on programs of career education. However, as project activities 
have progressed, especial 1y' field testing, it has been ^learned that 'the 
activities can easily be adapted to all areas of ediucational programs. 
Plans to disseminate the project outcomes ort a nationwide basis thnough 
a commercial publisher have not been completed but hopefully in the near^ 

* :r * . . 

fuJLure these plans can be made. 
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